In the field of machine learning, acquiring training data for a classifier presents a challenge. A core aspect of machine learning is classification, or the task of writing a computer program to differentiate between two things. Just as humans learn to do this via example, machine learning algorithms are presented with examples called training data or labels. Classifiers are used to rewrite queries for search engines. If a user types a word into a query box, a similar word is used to match relevant documents containing the original word or the similar word. However, collecting training requires significant time and money. Often, this is accomplished by paying human judges to spend considerable time inspecting examples. Accordingly, a more efficient manner of building a classifier and determining acceptable query rewrites is needed.